{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b6aa1234",
   "metadata": {},
   "source": [
    "# Machine Learning in Engineering (EE658) - Spring 2021\n",
    "## Assignment #3 -  Neural Network\n",
    "### DUE DATE:  Friday, April 9th, 2021"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f042c264",
   "metadata": {},
   "source": [
    "# Predicting the patients' healthy by using Neural Network\n",
    "## For this assignment, we are asked to implement a neural network using one hidden layer that can be used to classify patients at risk for breast cancer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b100d87",
   "metadata": {},
   "source": [
    "## (a) Load the data from disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "8d6ab64d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df = pd.read_csv(\"breast-cancer-wisconsin.data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "5a1541e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1000025</th>\n",
       "      <th>5</th>\n",
       "      <th>1</th>\n",
       "      <th>1.1</th>\n",
       "      <th>1.2</th>\n",
       "      <th>2</th>\n",
       "      <th>1.3</th>\n",
       "      <th>3</th>\n",
       "      <th>1.4</th>\n",
       "      <th>1.5</th>\n",
       "      <th>2.1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1002945</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1015425</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1016277</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1017023</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1017122</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   1000025  5   1  1.1  1.2  2  1.3  3  1.4  1.5  2.1\n",
       "0  1002945  5   4    4    5  7   10  3    2    1    2\n",
       "1  1015425  3   1    1    1  2    2  3    1    1    2\n",
       "2  1016277  6   8    8    1  3    4  3    7    1    2\n",
       "3  1017023  4   1    1    3  2    1  3    1    1    2\n",
       "4  1017122  8  10   10    8  7   10  9    7    1    4"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7d1d9fe",
   "metadata": {},
   "source": [
    "## (b) Refer to the patients features data by \"X\", and refer to the label feature (Class) by \"y\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "1490d237",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.set_index(df['1000025'],inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "1bb63ef9",
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = set(df.columns)\n",
    "y = df['2.1'].map({2:0, 4:1})\n",
    "columns.remove('2.1')\n",
    "person_id = df['1000025']\n",
    "columns.remove('1000025')\n",
    "X = df[columns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "18400e55",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>1.3</th>\n",
       "      <th>1.2</th>\n",
       "      <th>3</th>\n",
       "      <th>1.5</th>\n",
       "      <th>5</th>\n",
       "      <th>2</th>\n",
       "      <th>1.4</th>\n",
       "      <th>1.1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000025</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1002945</th>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1015425</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1016277</th>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1017023</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1017122</th>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          1  1.3  1.2  3  1.5  5  2  1.4  1.1\n",
       "1000025                                      \n",
       "1002945   4   10    5  3    1  5  7    2    4\n",
       "1015425   1    2    1  3    1  3  2    1    1\n",
       "1016277   8    4    1  3    1  6  3    7    8\n",
       "1017023   1    1    3  3    1  4  2    1    1\n",
       "1017122  10   10    8  9    1  8  7    7   10"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "9972816d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000025\n",
       "1002945    0\n",
       "1015425    0\n",
       "1016277    0\n",
       "1017023    0\n",
       "1017122    1\n",
       "Name: 2.1, dtype: int64"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4df0cc6a",
   "metadata": {},
   "source": [
    "## (c) Split the dataset into two subsets: training and testing datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "687edbbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "3e522612",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(X,y,test_size=0.2, random_state=44)\n",
    "scaler = StandardScaler()\n",
    "x_train = scaler.fit_transform(x_train)\n",
    "x_test = scaler.fit_transform(x_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "1d6d468e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.15582057,  1.61513871, -0.37914847, ..., -0.20376776,\n",
       "        -0.67950861,  0.43878543],\n",
       "       [-0.15582057, -0.7457369 , -0.37914847, ..., -0.20376776,\n",
       "        -0.67950861, -0.18486899],\n",
       "       [ 0.4586831 ,  1.61513871,  1.5165939 , ...,  1.11288547,\n",
       "         0.22363575,  1.06243985],\n",
       "       ...,\n",
       "       [-0.15582057, -0.7457369 , -0.37914847, ..., -0.64265218,\n",
       "        -0.67950861, -0.80852341],\n",
       "       [ 1.99494228,  1.61513871,  1.5165939 , ...,  0.67400106,\n",
       "         1.42782823,  1.99792149],\n",
       "       [-0.77032425, -0.48341739, -0.69510554, ..., -0.64265218,\n",
       "        -0.67950861, -0.80852341]])"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "35bb0910",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "1bac56d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_1 = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(3, 2), random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "39368d11",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPClassifier(alpha=1e-05, hidden_layer_sizes=(3, 2), random_state=1,\n",
       "              solver='lbfgs')"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_1.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "599fea81",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_1 = clf_1.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "d9ffa645",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "8221cd59",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9571428571428572"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_test, y_pred_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "216af5df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9214285714285714"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_2 = MLPClassifier(solver='lbfgs', alpha=0.1, hidden_layer_sizes=(7, 2), random_state=1,max_iter=15000)\n",
    "clf_2.fit(x_train, y_train)\n",
    "y_pred_2 = clf_2.predict(x_test)\n",
    "accuracy_score(y_test, y_pred_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e03ac5d",
   "metadata": {},
   "source": [
    "## (d) Python implementation of the neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "7349c0bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "ed05e6e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "enc = OneHotEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "6db81eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "enc.fit(np.array(y_train).reshape(-1,1))\n",
    "y_t = enc.transform(np.array(y_train).reshape(-1,1)).toarray()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "04c4efae",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_t = y_t.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "b9811596",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_t = x_train.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "c3c5d522",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9 558 7 2 0.1 (9, 558) (2, 558)\n"
     ]
    }
   ],
   "source": [
    "#sigmoid\n",
    "def g1(t):\n",
    "    return 1/(1 + np.exp(-t))\n",
    "def g1_prime(t):\n",
    "    return g1(t)*(1-g1(t))\n",
    "# softmax\n",
    "def g2(x):\n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum(axis=0)\n",
    "#Defining the parameters\n",
    "n_x = x_t.shape[0]\n",
    "m = x_t.shape[1]\n",
    "n_hidden = 7\n",
    "n_y = 2\n",
    "alpha = 0.1\n",
    "print(n_x, m, n_hidden, n_y, alpha, x_t.shape, y_t.shape)\n",
    "\n",
    "def NN(X, y, n_x, n_hidden, n_y, alpha, iterations):\n",
    "    w_1 = np.random.randn(n_hidden, n_x)\n",
    "    b_1 = np.zeros((n_hidden, 1))\n",
    "    w_2 = np.random.randn(n_y, n_hidden)\n",
    "    b_2 = np.zeros((n_y, 1))\n",
    "    error_list = []\n",
    "    for i in range(iterations):\n",
    "         # Forward Prop\n",
    "        z_1 = np.dot(w_1, X) + b_1\n",
    "        a_1 = g1(z_1)\n",
    "        z_2 = np.dot(w_2, a_1) + b_2\n",
    "        a_2 = g2(z_2)\n",
    "\n",
    "         # Back Prop\n",
    "        dz_2 = a_2 - y;\n",
    "        dw_2 = np.dot(dz_2, a_1.T)/m\n",
    "        db_2 = np.sum(dz_2, axis=1, keepdims=True)/m\n",
    "        dz_1 = np.dot(w_2.T, dz_2) * g1_prime(z_1)\n",
    "        dw_1 = np.dot(dz_1, X.T)/m\n",
    "        db_1 = np.sum(dz_1, axis=1, keepdims=True)/m\n",
    "        w_2 -= alpha * dw_2\n",
    "        b_2 -= alpha * db_2\n",
    "        w_1 -= alpha * dw_1\n",
    "        b_1 -= alpha * db_1\n",
    "        error = -1.0/m * np.sum(\n",
    "              np.sum(y*np.log(a_2)) + \n",
    "              np.sum((1-y)*np.log(1-a_2))\n",
    "           )\n",
    "        error_list.append(error)\n",
    "    return w_1, b_1, w_2, b_2, error_list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "2c9a6e0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "w_1, b_1, w_2, b_2, error_list = NN(x_t, y_t, n_x, n_h, n_y, alpha, 15000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "9e2f87b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_t_test = x_test.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "96ee32b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "z1 = np.dot(w_1, x_t_test) + b_1\n",
    "a1 = g1(z1)\n",
    "z2 = np.dot(w_2, a1) + b_2\n",
    "a2 = g2(z2)\n",
    "pred = np.argmax(a2, axis=0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "88631d00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0,\n",
       "       1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0,\n",
       "       0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0,\n",
       "       0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0,\n",
       "       1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0,\n",
       "       1, 0, 0, 1, 1, 0, 1, 0], dtype=int64)"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "cf82ead0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9571428571428572"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(pred, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac7cb43f",
   "metadata": {},
   "source": [
    "## The result show that python implementation of the neural network method under the conditions of using the same parameters as sklearn NN function such as hidden neuron, learning rate and so on, which obtains more higher accuracy score. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60a1e8bf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
